Worked on document for Martin.
Solved issue with TPC using davix:
After another few hours of looking around (made difficult by the fact we don’t run the s3 endpoint so couldn’t see the GET request being passed to it) and reading through the TPC technical docs, it’s because the Host header isn’t being sent in the GET request from our storage manager (StoRM). I had to explicitly set it in the header of the COPY request for it to pass on, e.g.
davix-cp --copy-mode pull -H "TransferHeaderHost: rgw.cscs.ch" -H "Authorization: Bearer ${BEARER_TOKEN}" --trace header,body https://rgw.cscs.ch:443/rucio/testing/23/c7/399d83d6-6883-4959-9bfc-610c14551cba?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=... https://srcdev.skatelescope.org:443/storm/sa/test_s3_pull_2
Unfortunately I couldn’t get it to work with FTS still. Is this the correct command? Using the prefix s3 will sign the URL right (and I’ve previously given you the bucket creds I think…)?
[rbarnsley@src-dev ~]$ fts-rest-transfer-submit --access-token=${BEARER_TOKEN} -s https://fts3-pilot.cern.ch:8446/ "s3://rgw.cscs.ch:443/rucio/testing/23/c7/399d83d6-6883-4959-9bfc-610c14551cba" https://srcdev.skatelescope.org:443/storm/sa/c Job successfully submitted. Job id: 14772a4a-bb90-11ee-bd12-fa163e522b2d
I can’t see the FTS logs on pilot anymore, gives me an unauthorized, so I can’t debug this easily any further.
Asked Mihai.
Solved issue with certs not being detected by storm by naming them with the md5 hash (not openssl default!) and adding .namespace files with the correct DN format (order sensitive).
Now just getting 403s. Emailed Mihai.
Still need to do CERN travel request.
Helping Robert B., Francois and Chris S.
Amended astroquery client so accepts token as envvars. Note:
because of the bug in Indigo, the only way i can reliably change token audience is using a refresh token, but this invalidates the previous access token. i can't do this with a token exchange yet (which doesn't invalidate the previous token) there are two consequences of this:
Looked a bit more at s3, specifically cert hashing and namespaces. Still no joy.
Astroquery: Built package and tested. Updated docs. get_data now uses DM API not Datalink.
S3: Looked at this but couldn't get StoRM to recognise DigiCert global (cscs root).
Changes to be made to astroquery: refresh token needs to change audience to whatever the previous token had if expired, change "No access token or refresh token are set, will not attempt token exchange" to imply both need to be set and change constructor so it accepts envvars (if,elif,else..) -> this actually needs to always set the paths if they're set as parameters, otherwise setting an access_token explictly won't persist...
Added download endpoint to DM API.
Added astroquery get_data endpoint and improved token ecosystem with persistence.
Need to update docs, build the pypi package and get feedback.
Token exchange introspects token. This isn't included in diagrams at the moment.
Actions:
Other bits:
https://confluence.skatelescope.org/display/SRCSC/MAG-824+AAI+API+chats+with+Purple
Prepping for API AAI discussions.
Adding get_data functionality but need DM API.
Spoke with Martin re: metadata SIG. Needs first draft (talk about what's been done re: engine, plugins, future works incl. benchmarking, getting use cases)
Need to try set up NLSRC and get back to Raymond.
Completed adding query_* functions and README.
Integrated sphinx docs into gitlab CI to generate gitlab page.
Started astroquery work, added a few functions.
Need to fix ADQL queries and add verification for radius/width/height units, then need to add get_data endpoints.
Added SODA defaults per convo with orange, awaiting confirmation it's OK.
Created a custom astroquery repo in src-service-apis and added astroquery as upstream. Added CI to build pypi package locally.
Tried a few more S3 commands and sent email to FTS team:
First I (successfully) upload a file to my S3 endpoint using the Rucio CLI:
export FILE=`uuidgen` && echo "test">> $FILE && rucio upload --rse CHSRC_S3 --lifetime 86400 --scope testing $FILE
2024-01-09 15:55:34,220 INFO Successfully uploaded file f8b6afef-7b6d-4f39-bf46-ee6662bfeb2dThen, I get the signed s3 url via the Rucio CLI:
rucio list-file-replicas testing:$FILE
CHSRC_S3: https://rgw.cscs.ch:443/rucio/testing/2d/5a/f8b6afef-7b6d-4f39-bf46-ee6662bfeb2d?X-Amz-Algorithm=...&X-Amz-Credential=...&X-Amz-Date=...&X-Amz-Expires=...&X-Amz-SignedHeaders=host&X-Amz-Signature=...Then, using gfal, I attempt to pull this file to a StoRM endpoint (I’ve exported my bearer token that is authorised for this storage to the env):
gfal-copy -vv https://rgw.cscs.ch:443/rucio/testing/2d/5a/f8b6afef-7b6d-4f39-bf46-ee6662bfeb2d?X-Amz-Algorithm=...&X-Amz-Credential=...&X-Amz-Date=...&X-Amz-Expires=...&X-Amz-SignedHeaders=host&X-Amz-Signature=...
https://srcdev.skatelescope.org:443/storm/sa/test_s3_pull_2 --copy-mode pullbut it fails with:
INFO Trying copying with mode 3rd pull
event: [1704816292879] BOTH http_plugin TRANSFER:TYPE 3rd pull
INFO Event triggered: BOTH http_plugin TRANSFER:TYPE 3rd pull
INFO Performing a HTTP third party copy
INFO Using bearer token for HTTPS request authorization
INFO Davix: Hop: https://srcdev.skatelescope.org:443/storm/sa/test_s3_pull_2
WARNING Copy failed with mode 3rd pull: copy HTTP 400 : Server Error
INFO Using bearer token for HTTPS request authorization
event: [1704816292902] BOTH http_plugin TRANSFER:EXIT ERROR: Copy failed (3rd pull). Last attempt: copy HTTP 400 : Server Error
INFO Event triggered: BOTH http_plugin TRANSFER:EXIT ERROR: Copy failed (3rd pull). Last attempt: copy HTTP 400 : Server Error
gfal-copy error: 112 (Host is down) - TRANSFER ERROR: Copy failed (3rd pull). Last attempt: copy HTTP 400 : Server ErrorIn my StoRM server logs I get some complaint about the “Credential” header being set to “gridsite” (I presume this is incorrect?):
2024-01-09T14:45:51.602Z [6f2136cc-927c-44a0-8526-6d066054c5e1] INFO 7269 --- [thread-pool-101576] o.i.s.webdav.tpc.TransferFilterSupport : Invalid request: Unsupported Credential header value: gridsiteEmails, expenses, slack messages etc.